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Parameter Sensitivity Analysis 

Estimation of the sensitivity of problem functions with respect to problem variables forms the basis 
for many of our modem day algorithms for engineering optimization. The most common application of 
problem sensitivities has been in the calculation of objective function and constraint partial derivatives for 
determining search directions and optimality conditions. A second form of sensitivity analysis, parameter 
sensitivity, has also become an important topic in recent years. By parameter sensitivity, we refer to the 
estimation of changes in the modeling functions and current design point due to small changes in the fixed 
parameters of the formulation. Methods for calculating these derivatives have been proposed by several 
authors (Armacost and Fiacco 1974, Sobieski et al 1981, Schmit and Chang 1984, and Vanderplaats and 
Yoshida 1985). Two drawbacks to estimating parameter sensitivities by current methods have been: (1) the 
need for second order information about the Lagrangian at the current point, and (2) the estimates assume no 
change in the active set of constraints. This paper addresses the first of these two problems and proposes a 
new algorithm that does not require explicit calculation of second order information. 


The estimation of changes in the modeling functions and 
design point due to small changes in the fixed parameters 

of the formulation. 
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Standard Form of NLP Parameter Sensitivity Problem 

To provide a framework about which to address the problem of parameter sensitivity analysis, we 
state the following standard form of the nonlinear programming problem which explicitly represents the 
problem parameters. 

In the formulation given, we assume that the problem functions f, g, and h can be either linear or 
nonlinear functions of the design variables but we are concerned primarily with the nonlinear case. We 
assume that the problem parameters p, are held fixed during the course of the optimization, and the optimal 
solution point, x* , satisfies the first order Kuhn-Tucker optimality conditions. 


Minimize 
Subject to 


OBJECTIVE: For a given p, find x% that satisfies the 

above problem. We are then interested in the effects of 
variations in p on the optimum. 

Given: pi’ = pi + Api 

Find: f(x*,p), x new 


f(x,p) 

hi(x,p) = 0 1 = 1,L 

gj(x,p) >0 j = 1,J 
Xmin — X < Xmax 
X = (Xi, X 2 , ... ,X n ) 

P = (Pi, P2, Pk) 
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Required Formulas 

For any change in the parameter Api, the new optimum value of the objective function or design 
variables can be estimated from the following linear extrapolations: 

Extrapolations based on these equations are bounded by the assumption that the active set remains 
the same. 


f(x*,p') = f(x*old) + Api 


df* 

dpi 


7 ) v ^ 

x*new = x*old + Api ^7 


where 


df* df v , a_h_L I . M 

dpi "api + ,2*/ api ' j“! Uj api 

df!__ JL afT ax* 

dpi api + dx api 

Derivatives to be determined: 


9f 3x* 3hj dgj 

3 p i’ dpi’ dpi’ 3pi 
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Methods for Calculating Parameter Sensitivities 
The Brute Force Method 


The brute force method is probably the most common method used to study the effect of problem 
parameters on solutions. The method is simply to change the parameter and then reoptimize the problem 
with the new value. This of course gives the truest indication of the effect of the parameter on the solution. 
A variation of the brute force method was proposed by Armacost and Fiacco (1974) and McKeown (1980) 
to calculate parameter sensitivities based on the central difference approximation given below. 


Given the incremental change A in pi , reoptimize the 
original problem at the new value of p. The sensitivity 
derivatives are given by the difference formulas. 

df* f(x*,pj + Apj) - f(x*,pj - Apj) 
dpi “ 2 Ap i 


8x* x*(pj + Apj) - x*(pj - Apj) 
3pi ” 2Api 


677 


Methods for Calculating Parameter Sensitivities 

Kuhn-Tucker Method 

A more accurate estimate of the sensitivity derivatives can be found by differentiating the Kuhn- 
Tucker optimality conditions with respect to a parameter. We refer to this as die Kuhn-Tucker method. The 
set of Kuhn-Tucker sensitivity equations have been derived independendy by several authors (Armacost and 
Fiacco 1974, McKeown 1980, Sobieski et al. 1981) and result in the following linear system of equations. 


Differentiate the Kuhn-Tucker conditions wrt to pi and 
solve the resulting linear system for the desired 
derivatives. 


2 .1 


r * n 

dx 


’ av*L ' 

V* L -V x (h,g) 


dpi 


3pi 



d(v,u) 

+ 

3(h,g) 

-Vx(h,g)T 0 


L dpi J 


_ 5pi - 
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Methods for Calculating Parameter Sensitivities 
Extended Design Space Method 

The final category of parameter sensitivity methods has been proposed by Vanderplaats 
(1984,1987). The method, known as the extended design space (EDS) method, is based on using feasible 
directions for estimating parameter sensitivity derivatives. The method extends the design space and solves 
the following subproblem to obtain the sensitivity derivatives. Both first and second order estimates have 
been developed for the method (Vanderplaats and Yoshida 1985). 


The fixed parameter pj is added to the set of design 
variables, 
x n+ i = Pi 

Solve the following subproblem for s, 
min Vx fT s 

st. gj+ V x gj T s> 0 j=l,J 
Calculate desired sensitivities from 

dx 

0p. — (Si, S2, ... »Sn )/ Sn + 1 

df df 9fT dx 
d p " 3p + dx dp 
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Assessment of Current Methods for Calculating Parameter 

Sensitivities 

As evidenced by their lack of extensive use, all the methods discussed above have some drawback 
associated with their use. Because the problem has to be reoptimized for several different values of the 
parameters, the efficiency of the Brute Force method is affected by the difficulty of the problem and the 
efficiency of the method used in the reoptimization. This approach is useful in studying large variations in 
parameters by plotting the response of the optimum versus the parameter, and has been used by Arbuckle 
and Sliwa (1984) and Robertson and Gabriele (1987). 

The Kuhn-Tucker method is also computationally expensive because it requires second derivatives 
of the objective function and the active constraints. For most engineering design problems, this type of 
information may be difficult to obtain. This method requires that the strict complementarity and linear 
independence assumptions hold at the optimal design. 

Finally, the first order ESD method is a very efficient, easy to implement method but it can provide 
inaccurate estimates of dx*/9p when the problem is not fully constrained and it does not provide 3u/9pi . 
The second order EDS method requires the calculation of second derivatives and also requires the solution 
of a quadratic approximating problem for each value of the parameter that is studied. However, the second 
order EDS method has the advantage of not being'affected by changes in the active set. 


Brute Force Method : Most commonly used method, 

provides accurate results, but inefficient. 

Kuhn-Tucker Method : Sound mathematical basis, but 

assumes no changes in the active set and requires second 
order information. 

Extended Design Space : Very efficient, easy to 

implement, but may not produce accurate estimates of 
dx/dp and does not provide du/dp 
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A Proposal for a New Method 

From the previous discussion, we can deduce that what is needed to improve current methods for 
parameter sensitivity analysis is an algorithm that does not require second derivatives, is able to accurately 
predict the sensitivity derivatives, and can calculate sensitivities at degenerate points. In this paper, we 
propose using a new algorithm based on the Recursive Quadratic Programming (RQP) method for 
accomplishing these goals. 

Our reasoning for such a method is based on the following virtues of the RQP method. In terms of 
number of function evaluations, the RQP method appears to be one of the most efficient methods available. 
This has been demonstrated in any of the published comparison studies in which codes for these methods 
were participants (Schittkowski, 1980 and Belegundu and Arora, 1985). Although the method is sensitive 
to variable and objective function scaling, it is not sensitive to constraint scaling. Finally, the RQP method 
provides an estimate of the Hessian of the Lagrangian, which can be useful for other purposes, and it is 
very efficient at locating an optimum, when the starting point is close to the true optimum. Both of these 
last advantages will be exploited in the development of our method for sensitivity estimation based on the 
RQP method. 


Proposal: 

Employ the Recursive Quadratic Programming Method 
(RQP) in conjunction with the Brute Force Method to 
estimate the required derivatives. 

Reasoning: 

The RQP method is very efficient when started near the 
optimum solution. 

If the RQP method is used to solve the original problem, 
an approximation of the Hessian of the Lagrangian will be 
available. 

Estimates of all derivatives, including duldp can be 
developed. 
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The Recursive Quadratic Programming Method 

All RQP methods use the same basic strategy of linearizing the constraints and approximating the 
Hessian of the Lagrangian to form a quadratic programming (QP) subproblem. The QP subproblem is then 
solved for the search direction s and a new estimate of the Lagrange multipliers of the constraints. The 
search direction s is then used to calculate a new estimate of the optimum. 

The step length a is determined by minimizing a line search penalty function P of the general form 

given below, where 12 represents some combination of the constraints and the Lagrange multipliers. The 
penalty function attempts to assure that both the objective function and the violation of the constraints are 
reduced. As the method converges, the step length a which minimizes P(x,u,v,R) approaches 1. 


Form the following subproblem to determine a search 
direction s 


Minimize 0.5 s T B s + s T Vf 
subject to VhTs + h = 0 
V gT s + g > 0 

Using s, perform a linear search to determine a new 
estimate of x* by minimizing a penalty function of the 
following general form, 

P(x,u,v,R) = f(x) + R*ft(h,g,u,v) 
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Basic Flowchart of RQP Method 

The basic flowchart of the RQP method is given below. Several good implementations of the RQP 
method are available (Beltracchi and Gabriele 1987, Arora and Tseng 1987, Bartholomew-Biggs 1987, or 
Gill, et al. 1986). A more complete discussion of RQP methods can be found in (Beltracchi 1985 or 
Beltracchi and Gabriele 1988). 
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The RQP Sensitivity Algorithm 

The new algorithm is based on combining the simplicity of the brute force method with the 
efficiency of the RQP method. The two characteristics of the RQP method that we feel can be exploited for 
determining parameter sensitivities are (1) an approximation to the Hessian of the Lagrangian is developed, 
and (2) if this approximation is exact (or close) then the RQP method will quickly and efficiently solve the 
perturbed problem. In other words, if we can develop good Hessian approximations, the RQP method is 
equivalent to applying Newton’s method to solve the Kuhn-Tucker conditions for the perturbed problem, 
which may require only 1 or 2 iterations of RQP. The small number of iterations, coupled with the fact that 
the RQP method should require only a one step line search, should allow the reoptimizations to occur 
without the need for many function evaluations. 

The end result of combining the differencing equations and the RQP method is a means to estimate 
the parameter sensitivities without the need to calculate higher order derivatives, and without an excessive 
number of function evaluations. Based on the above arguments, we propose the following procedure to 
calculate parameter sensitivity derivatives. 


Step 0. Given an optimal solution x*, f*, u*, an active 
set of constraints, and an approximation to the 
Hessian of the Lagrangian, all achieved by 
convergence of the RQP method (using the 
SR1/PD/BFS update). 

Step 1. Perturb the fixed parameter p< to p,* = pi® + Ap, 

where Apiis some small perturbation to p* 

Step 2. Perform two complete iterations of the RQP 
method. During the RQP iterations, update the 
Hessian approximation. From the results of the 
RQP iterations obtain f+, x*, u+, and gj* j <t active 
set. 

Step 3. Perturb the fixed parameter p s to pr = pi®- Api 

3x 

Step 4. Start from x 0 = x*(p f o) - ^-Api 

where 3x/3pjis calculated using forward 
differencing approximations and x*(pi») and x+(pi° 
+ Ap) from step 2. 

Perform one complete iteration of the RQP 
method to find f-, x , u-, and gj- j £ active set. 

Step 5. Obtain estimates for the sensitivity derivatives 
from the following central difference 
approximations 

df* f+ - f- 3x * x + - x- 
df 2Ap ’ 0p “ 2Ap 


du* _ u + - u - 
3p “ 2Ap 
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Equivalence to the Kuhn-Tucker Method 

The major questions to be answered about the proposed algorithm are does it provide the desired 
sensitivities, and what are the possible sources of error. An investigation of the theoretical properties of the 

RQP based sensitivity algorithm reveals that in the limit, as Ap goes to zero, the new method provides an 
estimate to the solution of the equations given below. These equations are equivalent to the Kuhn-Tucker 
sensitivity equations with the Hessian of the Lagrangian replaced by an approximation B that is provided by 
the RQP method. The details of this derivation are too lengthy to be presented here but are included in 
(Beltracchi 1988). From the above, we can see that the new algorithm will provide accurate estimates of the 

parameter sensitivities if B is a good approximation of V x L, and the differencing formula is a good 
approximation of the equations below. 


It can be shown that the proposed method is equivalent to 
the following linear set of equations, 


B -V x g 
V x gT 0 

These equations are the Kuhn-Tucker equations with the 
Hessian Approximation B replacing L. 


~3x~ 


av x L 

3p 


ap 

du 

Lap-1 

+ 

ag. 

_3d _ 
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EFFICIENCY OF SENSITIVITY ALGORITHMS 


As a measure of the efficiency of the new method, we present a comparison of the number of 
function evaluations required by the competing algorithms. The graph below represents the number of 
function evaluations required by various algorithms to find the sensitivity of the first parameter to be 
studied. The results are plotted for various problem sizes where n is the number of design variables. 

Most of the work associated with the Kuhn-Tucker method is incurred when the Hessian matrix is 
calculated. For the first order EDS method, the work does not increase with problem size. However, as 
mentioned before, the method is not always accurate and 5u/9p is not determined. Each of the RQP 
methods is either more efficient or as efficient when compared to the Kuhn-Tucker method for small 
problems (n<5), and considerably more efficient for larger problems (n>5). 



Number of Design Variables 
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The SR1/PD/BFS Update 

A major concern of the new algorithm is how well the Hessian approximation delivered from the 
RQP method agrees with the true Hessian. Toward this end, considerable research was conducted to find 
a variable metric update that provides good Hessian approximations without degrading the performance of 
the RQP algorithm. The two leading candidates were the Broyden-Fletcher-Shanno (BFS) method, and the 
Symmetric Rank One (SRI) method. 

The SR1/PD/BFS (Symmetric Rank One/ Positive Definite/ Broyden Fletcher Shanno) is a hybrid 
variable metric update that combines the best features of the SRI and BFS updates (Beltracchi 1988). The 
SRI update has the advantage of not requiring exact line searches and producing good Hessian 
approximations, however it has the drawbacks of being undefined or producing indefinite Hessian 
approximations for some problems. The BFS update has the advantage of being self correcting; however, it 
has the drawback of requiring exact line searches, and the Hessian approximation does not converge unless 
fairly accurate line searches are performed. The SR1/PD/BFS update uses the BFS update when the SRI 
update is undefined or likely to produce an indefinite Hessian approximation. The PD stands for a positive 
definite check (implemented in step 4), used to insure the new Hessian approximation is positive definite. 
Testing in Beltracchi (1988) found the SR1/PD/BFS update produced the best Hessian approximations. 


(Symmetric Rank One / Positive Definite / Broyden Fletcher Shanno) 

1 . Calculate y T (By - z) 

2 . If abs(yT(By - z)) < 10 io goto step 7 

3. Calculate (J>*,<J>: 

x* (.y t z ) 2 

^ (yTz)2 . zTBz yTfi-ly 

a _ y T z 

^ yT z - zTB z 

4. If <J) < (J)* goto step 7. 

5 . If yTz < 0 and yT(By - z) < 0 goto step 7. 

6 . Update Hessian approximation using the SRI update 
and return. 

7 . Update Hessian approximation using the BFS update 
and return. 
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Number of Problems Solved 


PERFORMANCE OF NEW SR1/PD/BFS UPDATE 


This slide shows the performance of the new SR1/PD/BFS variable metric update when 
implemented within the RQP method on a set of 13 commonly used test problems. The method is compared 
with the BFS and the SRI method. We see from this plot that the new update provides the same level of 
robustness and efficiency as the BFS method and is generally more efficient than the SRI method. 



0 10 20 30 40 50 60 70 80 90 100 


Number of Iterations Required to Solve the Problem 
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Convergence of the Hessian Approximation for Various 
Updates in Broyden's Family 


This table shows the ability of the tested variable metric updates to approximate the Hessian of the 
Lagrangian on a set of test problems. Analytical Hessians were developed for each test problem and 
compared to the approximated Hessian returned from the RQP method. The entries in the table report the 
Frobenius norms of the difference between the known optimal Hessian, and the identity matrix, and with 
the final approximate Hessian. From this we see that the new SR1/PD/BFS update returned approximations 
that were as good or better than the BFS method in all cases, and better than the SRI method in all but one 
case. In this instance (Woods), the scaling of the objective function affected the final result. The last two 
rows demonstrate that scaling the objective allowed the new update to improve the approximation 
considerably. 


Problem 

IIHMIlF 

BFS 

SRI 

SR1/PD/BFS 

RTS01 

2.291 

1.396 

0.061 

0.0008 

RTS01/100 

1.377 

0.0183 

0.974 

0.00023 

RTS01*100 

369.12 

206.7 

0.243 

0.243 

RTS02 st pt 1 

7.384 

3.00 

3.0 

3.0 

RTS02 st pt 2 

7.384 

5.430 

0.0 

0.0 

RTS03 

15.93 

7.76 

0.105 

0.105 

RTS04 

23.28 

3.623 

9.307 

3.356 

Woods 

1352.46 

141.3 

17.17 

75.38 

Woods/100 

13.54 

1.908 

0.184 

0.0944 

Woods/1000 

1.394 

0.0283 

0.974 

0.113 
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TESTING OF RQP SENSITIVITY ALGORITHM 

INITIAL TEST SET 

Two phases were employed in testing the new algorithm. The first phase involved a test set of 
known characteristics, whose Hessians and sensitivity derivatives could be determined analytically. The 
initial test set involved 4 test problems with 3-4 parameters each. Various algorithm parameters were 

studied, such as the step size Ap, as well as the effectiveness of the SR1/PD/BFS update, the number of 
RQP iterations to allow for solving the perturbed problem, and the updating of the Hessian approximation 
during the sensitivity analysis. The table below provides a sample of the results obtained on this initial test 
set using the RQP based sensitivity algorithm. The entries represent the error in the sensitivity derivatives 
obtained from the new algorithm and the known sensitivities. The errors were calculated using the formulas 
established by Sandgren (1977). 

The results shown here were fairly typical of the results obtained on all the test problems in the initial 
test set. In general, the results are very good and certainly usable for engineering purposes. 


PROBLEM 1 PROBLEM 2 



Pi 

P 2 

P3 

PI 

P2 

df 

dp 

.0002(10-9) 

1.9(10-5) 

0.126(10-6) 

0.139(10-6) 

0.00 

ex 

0.00 

7.9(10-5) 

3.11(10-7) 

9.08(10-7) 

2.00(10-7) 

eu 

0.00 

2.66(10-3) 

1.55(10-6) 

9.79(10-7) 

1.43(10*9) 

£ g 




6.33(10-7) 

0.00 
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TESTING OF RQP SENSITIVITY ALGORITHM 
ENGINEERING TEST PROBLEMS 


The second phase of the testing consisted of applying the new algorithm to a set of engineering test 
problems where known sensmvi^es and Hessians could not be developed analytically. To determine the 
accuracy of the sensitivity derivatives returned by the new method, the actual sensitivities for each test 
problem were developed by reoptimizing the problem over a range of values for each parameter. The results 
were then fit with either a linear or quadratic curve, depending on the amount of nonlinearity present, and 
the resulting curve was used to estimate the derivatives. These were then compared with the derivatives 
obtained from the RQP based algorithm. 

This slide describes the three engineering test problems that were used. Complete descriptions are 
available in Beltracchi (1988). 


Four Bar Slider Crank Problem: Design a four bar slider 

crank mechanism to generate a desired coupler path. 

Four parameters were studied: a movability criteria 
parameter, two timing parameters, and the y position of a 
precision point. 

Weld Beam Problem: Design a welded beam structure for 

minimum cost. Parameters studied: fixed length of beam, 
load on beam, yield stress in beam, and allowable shear in 
weld. 

Corrugated Bulkhead: Design a ship bulkhead for 

minimum weight. Parameters studied: change in position 
of two stringers, and height of the free liquid. 
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ENGINEERING TEST PROBLEMS: RESULTS 

The following tables report on the accuracy of the parameter sensitivity derivatives returned by the 
RQP based algorithm for the engineering test problems. The following general conclusions can be drawn 

from these results: . 

• The method produces results for df/dp, dx/dp, and dg/dp that are in the range of 3-4 significant 

digits of accuracy. 

• The results for the four bar mechanism problem are generally worse than the other two 
problems. This is due to the highly nonlinear nature of this problem and the difficulty in 
locating accurate optimal points. 


Four Bar Slider Crank 


df 

dp 

Pi 

2.0(10-5) 

P2 

4.86(10-3) 

P 3 

2.25(10-5) 

P 4 

3.29(10-7) 

ex 

2.32(10-2) 

6.38(10-3) 

7.67(10-2) 

2.65(10-2) 

eu 

3.62(10-1) 

3.34(10-2) 

6.14(10-2) 

5.48(10-2) 

eg 

8.81(10-5) 

1.94(10-3) 

7.82(10-2) 

9.36(10-3) 

df 

dp 

PI 

1.26(10-5) 

Welded Beam 

P 2 P 3 

8.7(10-6) 8.62(10-6) 

P 4 

1.89(10-4) 

ex 

7.59(10*5) 

4.28(10-5) 

3.12(10-5) 

1.26(10-4) 

eu 

2.45(10-4) 

3.54(10-4) 

1.80(10-4) 

1.75(10-4) 

eg 

8.61(10-7) 

1.05(10-4) 

1.27(10-18) 

1.45(10-4) 


Bulkhead Problem 


df 

dp 

Pi 

P 2 

P 3 

3.21(10-5) 

3.34(10-6) 

3.51(10-5) 

ex 

5.21(10-5) 

2.81(10-5) 

1.92(10-4) 

eu 

7.74(10-5) 

1.08(10-3) 

8.11(10-5) 

eg 

5.12(10-5) 

2.69(10-5) 

3.23(10-5) 
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CONCLUSIONS 


In this paper, we have proposed an alternative to current methods for estimating parameter 
sensitivities. The new method is based on combining the use of an RQP algorithm with differencing 
formulas which provides a means to estimate the sensitivities without the need for calculating second order 
derivatives. The method has been tested against two different test sets, one with analytical derivatives 
available and one without, and in both cases the method was able to accurately determine the sensitivity 
derivatives. The two major issues in implementing the algorithm concern the ability to formulate an accurate 
approximation to the Hessian of the Lagrangian and the ability to accurately estimate the modified Kuhn- 
Tucker sensitivity equations using the differencing formulas. Based on the testing performed so far, we arc 
led to the following conclusions: 

1 . In terms of efficiency, the method is competitive with existing methods. 

2 . Parameter sensitivity analysis can be performed using the RQP based method. 

3 . The Hessian approximation is improved if updating is allowed during the sensitivity calculations. 

4. The SR1/PD/BFS update in general provided more accurate estimates of the Hessian of the 
Lagrangian than either the BFS or SRI updates on our test set. The initial testing of this update was 
very encouraging in terms of the convergence of the Hessian approximation to the true Hessian. 


1. The method is competitive with existing methods. 

2. Parameter sensitivity analysis can be performed using the 
RQP based method. 

3. The Hessian approximation is improved if updating is 
allowed during the sensitivity calculations. 

4. The SR1/PD/BFS update in general provided more 
accurate estimates of the Hessian of the Lagrangian than 
either the BFS or SRI methods. 
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